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APPLIED AND REAL NEURAL NETWORKS: 

A COORDINATED AND INTERDEPENDENT INVESTIGATION OF BOTH 

Abstract 

One and a half years ago, Caltech organized a new graduate program in 
Computation and Neural Systems (CNS) . This program involves 15 faculty 
members with interests as diverse as statistical physics, concurrent 
computing, analog VLSI, signal processing, optical computing, machine vision 
robotics, and the neurob i ol og i ca 1 study of numerous real neural systems. 

The Bower laboratory is an integral part of the CNS program with our primary 
interest being the coordinated study of information processing in real 
neural networks. The principal approach taken is one that views these 
networks as systems of complex processing elements having functions that are 
intimately related to their specific distributed architectures. Within the 
lab, our multidisciplinary approach includes standard anatomical and 
physiological investigations linked to computer modeling techniques. In 
addition, we are developing new experimental techniques which directly 
address computational issues in real neural structures. For example, we are 
using modern silicon manufactur i ng technology to make multisite brain 
recording electrodes which capture the activity of multiple, functionally 
related neurons. We have also constructed a general-purpose neural network 
simulator with interactive graphics (CAD/CAM for neural networks) that runs 
on concurrent computers. Finally, we have been exploring the use of applied 
neural networks in recognizing and categorizing recorded signals from real 
neurons. Each of these efforts is described. This work is sponsored by the 
Whitaker Foundation, the Joseph Drown Foundation, and Lockheed Corporation. 
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ABSTRACT 

Based on anatomical and physiological data, we have developed a computer simulation of piri- 
form (olfactory) cortex which is capable of reproducing spatial and temporal patterns of actual 
cortical activity under a variety of conditions. Using a simple Hebb-type learning rule in conjunc- 
tion with the cortical dynamics which emerge from the anatomical and physiological organiza- 
tion of the model, the simulations are capable of establishing cortical representations for differ- 
ent input patterns. The basis of these representations lies in the interaction of sparsely distribut- 
ed, highly divergent/convergent interconnections between modeled neurons. We have shown that 
different representations can be stored with minimal interference, and that following learning 
these representations are resistant to input degradation, allowing reconstruction of a representa- 
tion following only a partial presentation of an original training stimulus. Further, we have 
demonstrated that the degree of overlap of cortical representations for different stimuli can 
also be modulated. For instance similar input patterns can be induced to generate distinct cortical 
representations (discrimination), while dissimilar inputs can be induced to generate overlapping 
representations (accommodation). Both features are presumably important in classifying olfacto- 
ry stimuli. 


INTRODUCTION 

Piriform cortex is a primary olfactory cerebral cortical structure which receives 
second order input from the olfactory receptors via the olfactory bulb (Fig. 1). It 
is believed to play a significant role in the classification and storage of olfactory 
information 1,2,3 . For several years we have been using computer simulations as a 
tool for studying information processing within this cortex 4,5 . While we arc ulti- 
mately interested in higher order functional questions, our first modeling objective 
was to construct a computer simulation which contained sufficient neurobiological 
detail to reproduce experimentally obtained cortical activity patterns. We believe 
this first step is crucial both to establish correspondences between the model and 
the cortex, and to assure that the model is capable of generating output that can 
be compared to data from actual physiological experiments. In the current case, 
having demonstrated that the behavior of the simulation at least approximates 
that of the actual cortex 4 (Fig. 3), we are now using the model to explore the 
types of processing which could be carried out by this cortical structure. In partic- 
ular, in this paper we will describe the ability of the simulated cortex to store and 
recall cortical activity patterns generated by stimulus various conditions. We 
believe this approach can be used to provide experimentally testable hypotheses 
concerning the functional organization of this cortex which would have been diffi- 
cult to deduce solely from neurophysiological or neuroanatomical data. 



Fig. 1. Simplified block diagram of the olfactory system and closely related structures. 


MODEL DESCRIPTION 

This model is largely instructed by the neurobiology of piriform cortex 3 . Axon- 
al conduction velocities, time delays, and the general properties of neuronal inte- 
gration and the major intrinsic neuronal connections approximate those currently 
described in the actual cortex. However, the simulation reduces both the number 
and complexity of the simulated neurons (see below). As additional information 
concerning the these or other important features of the cortex is obtained it will be 
incorporated in the model. Bracketed numbers in the text refer to the relevent 
mathematical expressions found in the appendix. 

Neurons. The model contains three distinct populations of intrinsic cortical 
neurons, and a fourth set of cells which simulate cortical input from the olfactory 
bulb (Fig. 2). The intrinsic neurons consist of an excitatory population of pyrami- 
dal neurons (which are the principle neuronal type in this cortex), and two popula- 
tions of inhibitory intemeurons. In these simulations each population is modeled 
as 100 neurons arranged in a 10x10 array (the actual piriform cortex of the rat 
contains on the order of 10 6 neurons). The output of each modeled cell type con- 
sists of an all-or-none action potential which is generated when the membrane 
potential of the cell crosses a threshold [2.3]. This output reaches other neurons 
after a delay which is a function of the velocity of the fiber which connects them 
and the cortical distance from the originating neuron to each target neuron [2.0, 
2.4]. When an action potential arrives at a destination cell it triggers a conduc- 
tance change in a particular ionic channel type in that cell which has a characteris- 
tic time course, amplitude, and waveform [2.0, 2.1]. The effect of this conductance 
change on the transmembrane potential is to drive it towards the equilibrium 
potential of that channel. Na + , C1‘, and K + channels are included in the model. 
These channels are differentially activated by activity in synapses associated with 
different cell types (see below). 
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Fig. 2. Schematic diagram of piriform cortex showing an excitatory pyramidal cell and two 
inhibitory intemeurons with their local interactions. Circles indicate sites of synaptic modifia- 
bility. 


Connection Patterns. In the olfactory system, olfactory receptors project to the 
olfactory bulb which, in turn, projects directly to the piriform cortex and other olfac- 
tory structures (Fig. 1). The input to the piriform cortex from the olfactory bulb is 
delivered via a fiber bundle known as the lateral olfactory tract (LOT). This fiber 
tract appears to make sparse, non-topographic, excitatory connections with pyra- 
midal and feedforward inhibitory neurons across the extent of the cortex 3,6 . In the 
model this input is simulated as 100 independent cells each of which make ran- 
dom connections (p=0.05) with pyramidal and feedforward inhibitory neurons 
(Fig. 1 and 2). 

In addition to the input connections from the olfactory bulb, there is also an 
extensive set of connections between the neurons intrinsic to the cortex (Fig. 2). 
For example, the association fiber system arises from pyramidal cells and makes 
sparse, distributed excitatory connections with other pyramidal cells all across the 
cortex 7,8,9 . In the model these connections are randomly distributed with 0.05 
probability. In the model and in the actual cortex, pyramidal cells also make exci- 
tatory connections with nearby feedforward and feedback inhibitory cells. These 
intemeurons, in turn, make reciprocal inhibitory connections with the group of 
nearby pyramidal cells. The primary effect of the feedback inhibitory neurons is to 
inhibit pyramidal cell firing through a Cl' mediated current shunting mecha- 
nism 10,11,12 . Feedforward intemeurons inhibit pyramidal cells via a long latency, 
long duration, K + mediated hyperpolarizing potential 12,13 . Pyramidal cell axons 
also constitute the primary output of both the model and the actual piriform cor- 


Synaptic Properties and Modification Rules. In the model, each synaptic con- 
nection has an associated weight which determines the peak amplitude of the con- 
ductance change induced in the postsynaptic cell following presynaptic activity 
[2.0]. To study learning in the model, synaptic weights associated with some of 
the fiber systems are modifiable in an activity-dependent fashion (Fig. 2). The 
basic modification rule in each case is Hebb-like; i.e. change in synaptic strength 
is proportional to presynaptic activity multiplied by the offset of the postsynaptic 
membrane potential from a baseline potential. This baseline potential is set 
slightly more positive than the Cl' equilibrium potential associated with the shunt- 
ing feedback inhibition. This means that synapses activated while a destination 
cell is in a depolarized or excited state are strengthened, while those activated 
during a period of inhibition are weakened. In the model, synapses which follow 
this rule include the association fiber connections between excitatory pyramidal 
neurons as well as the connections between inhibitory neurons and pyramidal neu- 
rons. Whether these synapses are modifiable in this way in the actual cortex is a 
subject of active research in our lab. However, the model does mimic the actual 
synaptic properties associated with the input pathway (LOT) which we have 
shown to undergo a transient increase in synaptic strength following activation 
which is independent of postsynaptic potential 15 . This increase is not permanent 
and the synaptic strength subsequently returns to its baseline value. 

Generation of Physiological Responses. Neurons in the model are represented 
as first-order "leaky" integrators with multiple, time- varying inputs [1.0]. During 
simulation runs, membrane potentials and currents as well as the time of 
occurence of action potentials are stored for comparison with actual data. An 
explicit compartmental model (5 compartments) of the pyramidal cells is used to 
generate the spatial current distributions used for calculation of field potentials 
(evoked potentials, EEGs) [3.0, 4.0]. 

Stimulus Characteristics. To compare the responses of the model to those of 
the actual cortex, we mimicked actual experimental stimulation protocols in the 
simulated cortex and contrasted the resulting intracellular and extracellular 
records. For example, shock stimuli applied to the LOT are often used to elicit 
characteristic cortical evoked potentials in vivo 16,17,18 . In the model we simulated 
this stimulus paradigm by simultaneously activating all 100 input fibers. Another 
measure of cortical activity used most successfully by Freeman and colleagues 
involves recording EEG activity from piriform cortex in behaving animals 19,20 . 
These odor-like responses were generated in the model through steady, random 
stimulation of the input fibers. 

To study learning in the model, once physiological measures were established, 
it was required that we use more refined stimulation procedures. In the absence of 
any specific information about actual input activity patterns along the LOT, we 
constructed each stimulus out of a randomly selected set of 10 out of the 100 input 


fibers. Each stimulus episode consisted of a burst of activity in this subset of 
fibers with a duration of 10 msec at 25 msec intervals to simulate the 40 Hz peri- 
odicity of the actual olfactory bulb input. This pattern of activity was repeated in 
trials of 200 msec duration which roughly corresponds to the theta rhythm period- 
icity of bulbar activity and respiration 21 ' 22 . Each trial was then presented 5 times 
for a total exposure time of 1 second (cortical time). During this period the Hebb- 
type learning rule could be used to modify the connection weights in an activity- 
dependent fashion. 

Output Measure for Learning. Given that the sole output of the cortex is in the 
form of action potentials generated by the pyramidal cells, the output measure of 
the model was taken to be the vector of spike frequency for all pyramidal neurons 
over a 200 msec trial, with each element of the vector corresponding to the firing 
frequency of a single pyramidal cell. Figures 5 through 8 show the 10 by 10 array 
of pyramidal cells. The size of the box placed at each cell position represents the 
magnitude of the spike frequency for that cell. To evaluate learning effects, overlap 
comparisons between response pairs were made by taking the normalized dot 
product of their response vectors and expressing that value as a percent overlap 
(Fig. 4). 
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Fig. 3. Simulated physiological responses of the model compared with actual cortical respons- 
es. Upper Simulated intracellular response of a single cell to paired stimulation of the input 
system (LOT) (left) compared with actual response (right) (Haberly & Bower,’84). Middle: 
Simulated extracellular response recorded at the cortical surface to stimulation of the LOT 
(left), compared with actual response (right) (Haberly,’73b). Lower Sbmulated EEG 
response reconed at the conical surface to odor-like input (left), for actual EEG see Freeman 
1978. 


Computational Requirements. All simulations were carried out on a Sun 
Microsystems 3/260 model microcomputer equipped with 8 Mbytes of memory and 
a floating point accelerator. Average time for a 200 msec simulation was 3 cpu 
minutes. 


RESULTS 


Physiological Responses 


As described above, our initial modeling objective was to accurately simulate 
a wide range of activity patterns recorded, by ourselves and others, in piriform 
cortex using various physiological procedures. Comparisons between actual and 
simulated records for several types of response are shown in figure 3. In general, 
the model replicated known physiological responses quite well (Wilson et al in 
preparation describes, in detail, the analysis of the physiological results). For 
example in response to shock stimulation of the input pathway (LOT), the model 
reproduces the principle characteristics of both the intracellular and location- 
dependent extracellular waveforms recorded in the actual cortex 9,17,18 (Fig. 3). 
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Fig. 4 . Convergence of the cortical response during training with a single stimulus with synaptic 
modification. 
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Fig. 5. Reconstruction of cortical response patterns with partially degraded stimuli. Left: 
Response, before training, to the full stimulus (left) and to the same stimulus with 50% of the 
input fibers inactivated (right). There is a 44% degradation in the response. Right: Response 
after training, to the full stimulus (left), and to the same stimulus with 50% of the input 
fibers inactivated (right). As a result of training, the degradation is now only 20%. 
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Fig. 6. Storage of multiple patterns. Left: Response to stimulus A after training. Middle: 
Response to stimulus B after training on A followed by training on B. Right: Response to 
stimulus A after training on A followed by training on B. When compared with the original 
response (left) there is an 85% congruence. 

Further, in response to odor-like stimulation the model exhibits 40 Hz oscillations 
which are characteristic of the EEG activity in olfactory cortex in awake, behaving 
animals 19 . Although beyond the scope of the present paper, the simulation also 
duplicates epileptiform 9 and damped oscillatory 16 type activity seen in the cortex 
under special stimulus or pharmacological conditions 4 . 

Learning 

Having simulated characteristic physiological responses, we wished to 
explore the capabilities of the model to store and recall information. Learning in 
this case is defined as the development of a consistent representation in the activ- 
ity of the cortex for a particular input pattern with repeated stimulation and synap- 
tic modification. Figure 4 shows how the network converges, with training, on a 
representation for a stimulus. Having demonstrated that, we studied three proper- 
ties of learned responses - the reconstruction of trained cortical response patterns 
with partially degraded stimuli, the simultaneous storage of separate stimulus 
response patterns, and the modulation of cortical response patterns independent 
of relative stimulus characteristics. 

Reconstruction of Learned Cortical Response Patterns with Partially Degrad- 
ed Stimuli. We were interested in knowing what effect training would have on the 
sensitivity of cortical responses to fluctuations in the input signal. First we pre- 
sented the model with a random stimulus A for one trial (without synaptic modifi- 
cation). On the next trial the model was presented with a degraded version of A 
in which half of the original 10 input fibers were inactivated. Comparison of the 
responses to these two stimuli in the naive cortex showed a 44% variation. Next, 
the model was trained on the full stimulus A for 1 second (with synaptic modifica- 
tion). Again, half of the input was removed and the model was presented with the 
degraded stimulus for 1 trial (without synaptic modification). In this case the dif- 
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Fig. 7. Results of merging cortical response patterns for dissimilar stimuli. Left Response to 
stimulus A and stimulus B before training. Stimuli A and B do not activate any input fibers in 
common but still have a 27% overlap in cortical response patterns. Right Response to stimu- 
lus A and stimulus B after training in the presence of a common modulatory input El. The 
overlap in cortical response patterns is now 46%. 

ference between cortical responses was only 20% (Fig. 5) showing that training 
increased the robustness of the response to degradation of the stimulus. 

Storage of Two Patterns. The model was first trained on a random stimulus A 
for 1 second. The response vector for this case was saved. Then, continuing with 
the weights obtained during this training, the model was trained on a new non- 
overlapping (i.e. different input fibers activated) stimulus B. Both stimulus A and 
stimulus B alone activated roughly 25% of the cortical pyramidal neurons with 25% 
overlap between the two responses. Following the second training period we 
assessed the amount of interference in recalling A introduced by training with B 
by presenting stimulus A again for a single trial (without synaptic modification). 
The variation between the response to A following additional training with B and 
the initially saved reponse to A alone was less than 15% (Fig. 6) demonstrating 
that learning B did not substantially interfere with the ability to recall A. 

Modulation of Cortical Response Patterns. It has been previously demon- 
strated that the stimulus evoked response of olfactory cortex can be modulated by 
factors not directly tied to stimulus qualities, such as the behavioral state of the 
animal 1 - 20,23 . Accordingly we were interested in knowing whether the representa- 
tions stored in the model could be modulated by the influence of such a "state" 
input. 

One potential role of a "state" input might be to merge the cortical response 
patterns for dissimilar stimuli; an effect we refer to as accomodation. To test this 
in the model, we presented it with a random input stimulus A for 1 trial. It was 
then presented with a random input stimulus B (non-overlapping input fibers). 
The amount of overlap in the cortical responses for these untrained cases was 
27%. Next, the model was trained for 1 second on stimulus A in the presence of an 
additional random "state" stimulus El (activity in a set of 10 input fibers distinct 
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Before Training After Training 

Fig. 8. Results of differentiating cortical response patterns for similar stimuli. Left; 
Response to stimulus A and stimulus B before training. Stimuli A and B activate 75% of 
their input fibers in common and have a 77% overlap in cortical response patterns. Right: 
Response to stimulus A and stimulus B after training A in the presence of modulatory input 
El and training B with a different modulatory input E 2. The overlap in conical response pat- 
terns is now 45%. 


from both A and B). The model was then trained on stimulus B in the presence of 
the same "state" stimulus El. After training, the model was presented with stim- 
ulus A alone for 1 trial and stimulus B alone for 1 trial. Results showed that now, 
even without the coincident El input, the amount of overlap between A and B 
responses was found to have increased to 46% (Fig 7). The role of El in this case 
was to provide a common stimulus component during learning which reinforced 
shared components of the responses to input stimuli A and B. 

To test the ability of a state stimulus to induce differentiation of cortical 
response patterns for similar stimuli, we presented the model with a random input 
stimulus A for 1 trial, followed by 1 trial of a random input stimulus B (75% of the 
input fibers overlapping). The amount of overlap in the cortical responses for these 
untrained cases was 77%. Next, the model was trained for a period of 1 second on 
stimulus A in the presence of an additional random "state" stimulus El (a set of 
10 input fibers not overlapping either A or B). It was then trained on input stimu- 
lus B in the presence of a different random "state" stimulus E2 (10 input fibers not 
overlapping either A, B, or El) After this training the model was presented with 
stimulus A alone for 1 trial and stimulus B alone for 1 trial. The amount of overlap 
was found to have decreased to 45% (Fig 8). In this situation El and E2 provided 
a differential signal during learning which reinforced distinct components of the 
responses to input stimuli A and B. 

DISCUSSION 

Physiological Responses. Detailed discussion of the mechanisms underlying 
the simulated patterns of physiological activity in the cortex is beyond the scope 
of the current paper. However, the model has been of value in suggesting roles for 




specific features of the cortex in generating physiologically recorded activity. For 
example, while actual input to the cortex from the olfactory bulb is modulated into 
40 Hz bursts 24 , continuous stimulation of the model allowed us to demonstrate 
the model’s capability for intrinsic periodic activity independent of the comple- 
mentary pattern of stimulation from the olfactory bulb. While a similar ability has 
also been demonstrated by models of Freeman 25 , by studying this oscillating 
property in the model we were able to associate these oscillatory characteristics 
with specific interactions of local and distant network properties (e.g. inhibitory 
and excitatory time constants and trans -cortical axonal conduction velocities). 
This result suggests underlying mechanisms for these oscillatory patterns which 
may be somewhat different than those previously proposed. 

Learning. The main subject of this paper is the examination of the learning 
capabilities of the cortical model. In this model, the apparently sparse, highly dis- 
tributed pattern of connectivity characteristic of piriform cortex is fundamental to 
the way in which the model leams. Essentially, the highly distributed pattern of 
connections allows the model to develop stimulus-specific cortical response pat- 
terns by extracting correlations from randomly distributed input and association 
fiber activity. These correlations are, in effect, stored in the synaptic weights of 
the association fiber and local inhibitory connections. 

The model has also demonstrated robustness of a learned cortical response 
against degradation of the input signal. A key to this property is the action of 
sparsely distributed association fibers which provide reinforcment for previously 
established patterns of cortical activity. This property arises from the modification 
of synaptic weights due to correlations in activity between in tra -cortical associa- 
tion fibers. As a result of this modification the activity of a subset of pyramidal 
neurons driven by a degraded input drives the remaining neurons in the response. 

In general, in the model, similar stimuli will map onto similar cortical respons- 
es and dissimilar stimuli will map onto dissimilar cortical responses. However, a 
presumably important function of the cortex is not simply to store sensory infor- 
mation, but to represent incoming stimuli as a function of the absolute stimulus 
qualities and the context in which the stimulus occurs. The fact that many of the 
structures that piriform cortex projects to (and receives projections from) may be 
involved in multimodal "state" generation 14 is circumstantial evidence that such 
modulation may occur. We have demonstrated in the model that such a modulato- 
ry input can modify the representations generated by pairs of stimuli so as to 
push the representations of like stimuli apart and pull the representations of dis- 
similar stimuli together. It should be pointed out that this modulatory input was 
not an "instructive" signal which explicitly directed the course of the representa- 
tion, but rather a "state" signal which did not require a priori knowledge of the 
representational structure. In the model, this modulatory phenomenon is a simple 
consequence of the degree of overlap in the combined (odor stimulus + modulator) 
stimulus. Both cases approached approximately 50% overlap in cortical responses 
reflecting the approximately 50% overlap in the combined stimuli for both cases. 


Of interest was the use of the model’s reconstructive capabilities to maintain the 
modulated response to each input stimulus even in the absence of the modulatory 
input. 

CAVEATS AND CONCLUSIONS 

Our approach to studying this system involves using computer simulation to 
investigate mechanisms of information processing which could be implemented 
given what is known about biological constraints. The significance of results pre- 
sented here lies primarily in the finding that the structure of the model and the 
parameter settings which were appropriate for the reproduction of physiological 
responses were also appropriate for the proper convergence of a simple, biologi- 
cally plausible learning rule under various conditions. Of course, the model we 
have developed is only an approximation to the actual cortex limited by our knowl- 
edge of its organization and the computing power available. For example, the 
actual piriform cortex of the rat contains on the order of 10 6 cells (compared with 
10 2 in the simulations) with a sparsity of connection on the order of p=0.001 
(compared with p=0.05 in the simulations). Our continuing research effort will 
include explorations of the scaling properties of the network. 

Other assumptions made in the context of the current model include the 
assumption that the representation of information in piriform cortex is in the form 
of spatial distributions of rate-coded outputs. Information contained in the spatio- 
temporal patterns of activity was not analyzed, although preliminary observation 
suggests that this may be of significance. In fact, the dynamics of the model itself 
suggest that temporally encoded information in the input at various time scales 
may be resolvable by the cortex. Additionally, the output of the cortex was 
assumed to have spatial uniformity, i.e. no differential weighting of information 
was made on the basis of spatial location in the cortex. But again, observation of 
the dynamics of the model, as well as the details of known anatomical distribution 
patterns for axonal connections, indicate that this is a major oversimplification. 
Preliminary evidence from the model would indicate that some form of hierarchical 
structuring of information along rostral/caudal lines may occur. For example it 
may be that cells found in progressively more rostral locations would have 
increasingly non-specific odor responses. 

Further investigations of learning within the model will explore each of these 
issues more fully, with attempts to correlate simulated findings with actual record- 
ings from awake, behaving animals. At the same time, new data pertaining to the 
structure of the cortex will be incorporated into the model as it emerges. 
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APPENDIX 


Somatic Integration 
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n c«itj “ number of cells in the simulation 

As ■ distance between adjacent cells 

d k « duration of conductance change due to input type k 

v, ■ velocity of signals for input type k 

£4 - latency for input type k 

p, « spatial attenuation factor for input type k 

p~" • minimum spatial attenuabon for input type k 

Ar, « refractory period 


Tj ■ threshold for cell j 
L.j - distance from cell i to cell j 
A , a distribution of synaptic density for input type k 
« synaptic weight from cell j to cell i 
gal') “ conductance due to input type k in cell i 
f t (r) a conductance waveform for input type k 
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n ctol « number of different channels per segment 
V„(t) - membrane potential of nth segment 
£•; “ membrane capacitance for segment n 
r* - axial resistance for segment n 
r' m membrane resistance for segment n 
g^(/) « conductance of channel c in segment n 
E c « equilibrium potential associated with channel c 
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I* (i ) ■ membrane current for segment n 
/„ » length of segment n 
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/?„ « membrane resistivity 
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